Skip to content

fix(recovery): keep the standby repair's isolation mark from being reverted - #4680

Merged
devantler merged 1 commit into
mainfrom
claude/standby-fence-field-manager-4673
Oct 9, 2026
Merged

devantler merged 1 commit into
mainfrom
claude/standby-fence-field-manager-4673

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

The reviewed repair for the broken wedding database replica stopped part-way on its first run. It marks the replica as isolated before touching its storage, and the platform's own Git synchronisation removed that mark seconds later because it did not recognise who had written it. The repair then saw the mark missing and held, so the replica is still down and production deploys that must restart its neighbours keep failing.

What

The repair now signs its changes with its own name, which the Git synchronisation leaves alone, so the isolation mark stays in place until the repair removes it. A new check fails if a repair step is ever sent without that name. Nothing in production changes until the repair is dispatched again from main.

Part of #4673

👉 After merge/promotion: dispatch the standby replacement once more from main; the previous attempt already made the old storage safe to keep.

… Flux

The first protected run fenced wedding-db-1 with a default kubectl patch.
kustomize-controller takes over fields owned by kubectl managers and removed
the annotation at its next reconciliation, 22 seconds before the repair's own
check held on the missing fence. Every patch now names its own field manager.

Part of #4673

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 3fe1c5dc-94df-4210-a389-4148685cace3
📥 Commits

Reviewing files that changed from the base of the PR and between 4622b50 and 492fe19.

📒 Files selected for processing (3)
  • scripts/replace-wedding-standby/README.md
  • scripts/replace-wedding-standby/main.go
  • scripts/replace-wedding-standby/main_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (1)
  • GitHub Check: 🔍 Detect Changes
🔇 Additional comments (3)
scripts/replace-wedding-standby/main.go (1)

140-143: LGTM!

Also applies to: 150-150

scripts/replace-wedding-standby/main_test.go (1)

543-571: LGTM!

scripts/replace-wedding-standby/README.md (1)

32-35: LGTM!


📝 Walkthrough

Walkthrough

The repair script now sets wedding-standby-repair as the field manager for patch commands. A new test checks the manager for PV, cluster, and PVC patches. The README documents how Flux reconciliation can remove a fence written with the default kubectl manager.

Priority: ⬆️ High

Merge Risk: ⚪ Minimal · up to 492fe

The repair’s fence writes use its dedicated field manager, and the reviewed changes show no uncovered path that would let reconciliation undo the fence. No actionable merge risk remains.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files. (1 skipped: 1 …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely describes the main change: preventing the standby repair's isolation mark from being reverted.
Description check Passed The description directly explains the failure, the field-manager fix, the added validation, and the required follow-up action.
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Evaluation at 492fe19ed7d43665ef15cad0a1ab40b4d5b7d018.

  • The repair cannot be exercised outside its protected main dispatch, so the evidence is the package's 67 tests passing with the race detector; the new one fails without the change and passes with it.
  • Live read-only evidence for the cause is on Wedding database replica fails PostgreSQL checkpoint recovery #4673: the Git synchronisation was the only recorded manager of the database object's metadata after re-applying it at 01:44:09Z, and the fence was gone 22 seconds before the hold.
  • CodeRabbit reviewed this head and raised nothing; no open review threads; checks green.

Not proven here: that the second dispatch completes. That is read from the run itself.

@devantler
devantler marked this pull request as ready for review October 9, 2026 02:41
@devantler
devantler added this pull request to the merge queue Oct 9, 2026
@devantler
devantler removed this pull request from the merge queue due to a manual request Oct 9, 2026
@devantler
devantler added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit e0c5558 Oct 9, 2026
34 checks passed
@devantler
devantler deleted the claude/standby-fence-field-manager-4673 branch October 9, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant